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Using Reliable Change to Calculate Clinically 
Significant Progress in Children with EBD: 

A BHRS Program Evaluation 

Vincent J. Thoder, James G. Hesky & Joseph D. Cautilli 
Abstract 

Children often have complex emotional and behavioral disorders (ADHD, ODD, Depression, PTSD, etc.). 
A large amount of research exists in the behavioral treatment of children with these disorders regarding specific 
behavioral problems. Much less research exists for the treatment of comprehensive problematic behaviors that these 
children experience in the real world. This effectiveness study evaluates the program at Behavior Analysis & 
Therapy Partners (BATP) for the treatment of children in their program with emotional and behavioral disorders. 
Sixteen children were treated in the community using what has come to be known as Behavioral Health 
Rehabilitative Services (BHRS) in the state of Pennsylvania. BATP uses a behavior analytic model for treating 
these disorders which features a functional behavioral assessment of problem behaviors and individual interventions 
based on the understanding of function. This evaluation found that 62.5% of the children made clinically significant 
reliable change within 278 days of entering the program. This paper presents a detailed analysis of those results. 
Since this is the first evaluation of the therapeutic properties of BHRS, clinical implications and future research 
directions are highlighted. 
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Emotional and behavioral disorders (EBD), as defined by Cook, Gresham, Kern, Barreras, 
Thorton, and Crews (2008), refers “to the full spectrum of students with social, emotional, and behavioral 
problems that do and do not receive special education services” (p. 132). The problematic behaviors can 
be of an internalizing or an externalizing characteristic - with children experiencing bouts of mild or 
clinical depression, attention deficit hyperactivity disorder (ADHD), conduct disorder (CD), oppositional 
defiant disorder (ODD), and even non-clinical externalizing behaviors - and, largely, educational 
programs of the past have been unsuccessful in helping these children thrive (Cook et al., 2008; Jull, 
2008). Children with EBD often have behavioral problems that lead to marked impairments in the 
academic performance (Nordess, 2005). On average, children with EBD receive lower grades and fail 
more courses with a drop-out rate exceeding 50% (Sacks & Kern, 2008). As of 2009, it was reported that 
students with EBD comprise 8% of all students with disabilities and the numbers are continuing to rise. 

Cook and colleagues (2008) confirm and expand on this claim by reporting the children with 
EBD are at a greater risk for poor school adjustment, for engaging in delinquent behaviors, and for adult 
psychopathology. This often leads the family to seek mental health services in the community. Creating 
a more pleasant and successful academic career is important, as children with EBD experience less than 
desirable social and economic outcomes, higher rates of unemployment, criminality, substance abuse, and 
aberrant sexual behavior (Sacks & Kern, 2008). Often times, the mental health system and the school 
system engage in treatment programs for these students. In Pennsylvania, this is referred to as behavioral 
health rehabilitative services (BHRS). 

Behavior Analysis & Therapy Partners (BATP) is a for-profit behavioral health agency whose 
clinicians serve a highly diverse population with respect to age (pre-school to elderly), diagnoses, 
presenting problems, and socio-economic status, including those with EBD. Depending on assignment, 
its BHRS division delivers treatment to children, many of whom have EBD, in the home, community, 
and/or school system. The BHRS are administered through a behavioral consulting process in which a 
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behavioral specialist consultant (BSC) works with a consultee (parent, teacher, or other socializing agent) 
to provide services to advance a formal problem-solving model that uses applied behavior analysis (ABA) 
intervention specification within a targeted behavior-ecology context (Bergan & Kratchowill, 1990). The 
BSC helps the child indirectly by affecting the consultee. Conversely, BATP offers the services of a 
mobile therapist (MT) and/or a therapeutic staff support (TSS) intervene in the child’s environment 
directly. According to Cautilli, Rosenwasser, and Clarke (2000), the MT affects behavior change by 
using behavioral therapy with both the individual child and the family while the TSSis a direct-care staff 
person who affects behavior change by providing proactive behavioral management and emotional 
support to the child. Both the TSS and the MT follow the treatment plan written by the BSC. 

BATP, like all BHRS, takes a family-focused approach in which both the families of the children 
and the children themselves are considered partners and experts with regards to the development and 
evaluation of services (Andersen-Butcher & Ashton, 2004). For BATP, this means the implementation of 
a parent training model and a use of siblings in the treatment, and the conjoing collaboration of the 
family, child, and treatment team on goal-setting. Rosen, Heckman, Cairo, and Burchard (1994) found 
that youths who receive wraparound services appear to be satisfied with the services and are less likely to 
act out when they feel involved and feel that their contingency-oriented care is unconditional. Services 
often use behavioral interventions to increase parental sensitivity to the needs and function of children’s 
behavior (Dunst and Kassow, 2008). In addition, Nordess (2005) found that wraparound services appears 
to adhere to the principles of individualized services, a family-driven approach, and unconditional care, 
consistent with BATP’s approach to service delivery. Rosen and colleagues (1994) suggest that this is 
provided evidence for the legitimacy of the principles of wraparound. 

Outpatient and school-based treatment programs have been shown to make statistically reliable 
and clinically significant improvements on symptom scales and to move to a less severe range of 
functioning (Cautilli, Harrington, Gillam, Denning, Helwig, Ettingoff, Valdes, & Angert, 2004; 

Karpenko, Owens, Evangelista, & Dodds, 2009; Mulick & Naugle, 2010; Wise, 2003) when pre- and 
post-treatment scales have been examined. Such comparisons provide a good starting point for the 
program evaluation of BHRS at BATP. The following analysis details the preliminary results of a select 
sample of such comparisons. 


Method 

Participants: To enter the program, children need to be between the ages of three and twenty-one 
and have a psychiatric diagnosis. Eor the purposes of this study, children who are diagnosed with any 
disorder on the Autism spectrum are excluded. In addition, children with developmental disabilities were 
only included if they have a primary diagnosis in the EBD category, such as anxiety, depression, 
oppositionality. Eor a detailed description of each child, please see Table 1. 

Description of Personnel. BATP subscribes to a behavior-analytic orientation and service model. 
At BATP, BSCs are required to be master’s level clinicians with an educational background in ABA; 

MTs are required to be master’s level clinicians with an education background in counseling and therapy. 
TSS workers are required, by the state of Pennsylvania, to have a “Bachelor’s degree in psychology, 
social work, counseling, sociology, education, criminal justice, or similar human service fields, with no 
previous work experience” (Commonwealth of Pennsylvania, 2006).’ 


Other qualifications can be found in the MA Bulletins 01-01-05, 29-01-03, 33-01-03, 41-01-02, 48-01-02, 
49-01-04, and 50-01-03 
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Training of Staff. Each staff member is provided with at least 15 hours of pre-service training and 
40 hours of on-going training to be completed throughout the calendar year (Adkins-Ruff, Cautilli, 

Clarke, & Thomas, 2001; Cautilli & Rosenwasser, 2001; Clarke & Cautilli, 2001; Thomas & Cautilli, 
2000; Weinberg, Cautilli, & Clarke, 2001). Weekly supervision is conducted by a board certified 
behavior analyst and Master’s level clinicians receive monthly supervision conducted by a licensed 
psychologist who is also a board certified behavior analyst. 

Interventions. Interventions are contingency oriented, designed to (1) eliminate antisocial and 
inappropriate behavioral patterns, (2) improve social relationships in the home and school milieus, and (3) 
increase social participation and engagement. 

Evaluation of Children. At BATE, each child is evaluated on entry and every four months by a 
licensed psychologist. As part of the evaluation process, parents are given the Child Behavior Checklist 
(CBCL) to rate their child’s behavioral problems. If the child is under the age of six years, the Child 
Behavior Checklist for Children Ages 1 */2 - 5 (CBCL/1.5-5) is administered; children and adolescents 
between the ages of 6 and 18 are administered the CBCL/6-18. For more information on the CBCL, 
please refer Achenbach & Rescorla (2000a) and Achenbach & Rescorla (2000b). 

Analytic Methods. To evaluate the program, these authors used the reliable change scores. 
Reliable change is important to understanding the effectiveness of a treatment program and it minimizes 
the statistical effect that may be unrelated to clinical significance (Cautilli, et ah, 2004; Eisen, 
Ramgamatjam. Sea;. & Spirp, 2007; Johnson, Dow, Lynch, & Hermann, 2006). Previous research 
(Jacobson, Roberts, Berns, &McGlinchey 1999; Jacobson, Follette, & Revenstorf, 1984; Jacobson & 
Truax, 1991) suggests a formula for calculating reliable change and reports that there are five possible 
outcomes stemming from the result of those calculations. For the purpose of this study recovery was sub- 
divided into two categories, “fully recovered” and “partially recovered.” Fully recovered refers to scores 
that ended in the normal range and met the 1 .96 reliable change score criteria. Partially recovered refers 
to scores that met the 1.96 reliable change score criteria and ended in the borderline range. “Unchanged” 
can be coded twice and refers to a score where neither criterion is met or where scores appear to be 
recovered or improved but do not show reliable change (Jacobson et. al, 1999). Regression occurs when 
the reliable change score is passed in the opposite direction (Jacobson et. al, 1999). 

Procedure. Every BATP case - both open and closed - was reviewed for possible inclusion in 
this program review. For inclusion criteria, refer to the “Participants” section. Additionally, each child 
had to have been administered the CBCL on at least two separate occasions. The CBCL had to have been 
completed by the same rater. 

The total number of cases that met the criteria was 16. The scores from the earliest and the most 
recent administration of the CBCL were collected. The scores were compared and the difference was 
observed. The difference was divided by the standard error of the measure obtained from the Manual for 
the ASEBA Preschool Eorms & Profiles and the Manual for the ASEBA School-Age Eorms & Profiles 
(Achenbach & Rescorla, 2000a; Achenbach & Rescorla, 2000b). 

Using the formula Jacobson and colleagues (Jacobson & Truax, 1991; Jacobson, Follette, & 
Revenstorf, 1984) developed, the scores obtained from the most recent administration of the CBCL were 
subtracted from the first administration. Next, that result was divided by the standard error of the 
measure given by Achenback & Rescorla (2000a; 2000b). The scores were compared to ±1.96 to obtain 
reliable change at the 95% confidence interval (Jacobson, Roberts, Berns, & McClinchey, 1999). 

If any change occurred on normal scales, the results were omitted. If any clinical significant 
progress was observed on a scale originally in the clinical (C) or the borderline clinical (B) range, the 
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scare was coded as a +1. If any clinically significant regression was observed on a scale that ended in the 
C or the B range, it was coded as a - 1 . The case was considered a success if the net result was a positive 
number. 


Treatment Integrity Checks. To check for treatment integrity, 75 treatment notes were drawn and 
reviewed at random. Treatment integrity refers to the notes mentioning the treatment goal as well as 
referencing at least one of the interventions as described in the treatment plan (Cautilli et. al, 2004). 

Typical Interventions. The treatment plans show a family-centered approach to treatment and are 
written on a developmental level that the family can be full partners in plan creation; they use a functional 
assessment, including both direct and indirect measures as proposed by Cone (1978); and they represent a 
comprehensive target area (Cautilli, Riley-Tillman, & Thomas, 2001). Each objective contains five 
components: a target person, identification of target behavior, identification of conditions under which 
the behavior is to be displayed, criteria for acceptable performance, and a timeline for achievement 
(Alberto & Troutman, 1999; Deno & Jenkins, 1967; Mager, 1962). 

Data 

Descriptive Statistics. Of the 16 subjects observed, 14 (87.5%) were male. Six (37.5%) of the 
subjects were African American and seven (43.75%) were Caucasian. The remaining three (18.75%) 
were bi-racial or unspecified. At the first administration of the CBCL, the average age of the child was 
nine years, three months. At the last available administration of the CBCL, the average age of the child 
was ten years, zero months. The average number of days between the administrations of the CBCL was 
approximately 278. All (100%) children received an individualized treatment plan based on a functional 
assessment. All (100%) 16 children’s treatment plans had a behavioral case formulation. Behavioral 
interventions were derived from evidence-based treatment practices and based on functional assessments 
for children with EBD. Interventions varied based on the child’s need, but included token economies, 
response cost, social skills training, antecedent control strategies, and contingency management. Parents 
and other community-based support systems received training on these interventions so that the 
appropriate level of skill could be transferred to the home setting. Consultation also occurred with the 
teachers and various school staff on behaviorally interventions to modify and manage the child’s 
behavioral challenges in school. 

Examining the primary diagnoses, ten (62.5%) children were diagnosed with ADHD; four (25%) 
were diagnosed with ODD; one (6.25%) was diagnosed with adjustment disorder with mixed disturbances 
of conduct and emotion; and one (6.25%) was diagnosed with post-traumatic stress disorder (PTSD). 
Secondary diagnoses ranged from ADHD to Mixed-Receptive Language Disorder and are included in 
Table 1. 


Table 1: Client Reliable Change (Both Positive and Negative) for Each Client 


Identifying 

Information 

Ages 
(Between 
Administratio 
ns of the 
CBCL) 

Days in 

Service 

(Between 

Administratio 

ns of the 

CBCL) 

Improved Scales 

Regressed 

Scales 

Scales with 
No Change 
or 

Clinically 
Insignifican 
t Change 

Net 

Scor 

e 

Client 1 

African 

American 

8 year, 8 
months - 9 
years, 8 
months 

378 

Social 

Competency (26- 
C, 40-N) 

Social 
Problems 
(72-C, 82- 
C) 

Attention 
Problems 
(70-C, 70- 
C) 

+3 
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Female 



Total Competence 
(26-C, 35-C) 

Internalizin 

Aggressive 


ADHD; 




g Problems 

Behavior 


Disruptive 



Rule-Breaking 

(50-N, 62- 

(75-C, 78- 


Behavior 



Behavior 

B) 

C) 


Disorder; 

Major 



(74-C, 71-C) 


Extemalizin 


Depressive 



Attention 


g Problems 


Disorder 



Deficit/Hyperactivi 


(75-C, 75- 


(Mild); 



ty Problems 


C) 


Specific 

Phobia 



(73-C, 70-C) 


Total 


(School); 



Oppositional 


Problems 


Mild MR 



Defiant Problems 


(71-C, 72- 


(By 



(70-C, 67-B) 


C) 


History) 





Conduct 
Problems 
(79-C, 77- 
C) 






Withdrawn 

School 






/ Depressed 

Competenc 






(66-B, 70- 

y 






C) 

(27-C, NC) 






Thought 

Total 






Problems 

Competence 


Client 2 




(58-N, 77- 
C) 

(25-C, NC) 
Attention 


Caucasian 

Male 




Rule- 

Problems 





Breaking 

(64-N, 66- 




Social 

Behavior 

B) 


ADHD; 

10 years, 3 


Competency 

(76-C, 79- 
C) 

Aggressive 

Behavior 


ODD; R/0 

months - 10 

142 

(25-C, 32-B) 

-4 

Bi-Polar 

Disorder; 

R/0 

Learning 

Disorder, 

NOS 

years, 8 
months 

Somatic Problems 

Internalizin 

(70-C, 75- 




(65-B, 61-N) 

g Problems 
(60-B, 66- 
C) 

C) 

Extemalizin 
g Problems 





Affective 

(74-C, 77- 






Problems 

C) 






(63-N, 70- 
C) 

Total 

Problems 






Oppositiona 

(70-C, 74- 






1 Defiant 

C) 






Problems 
(66-B, 73- 

Conduct 






C) 

Problems 
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(83-C, 86- 
C) 






Social 

Competence 
(34-B, 28- 

Total 






C) 

Competence 
(35-C, 37- 






Thought 

Problems 

B) 






(58-N, 70- 

Withdrawn 


Client 3 




C) 

/ Depressed 
(68-B, 68- 


African 



School 

Aggressive 

B) 


American 



Behavior 



Male 

6 years, 5 
months - 6 

207 

Competence 
(33-B, 40-N) 

(68-B, 79- 
C) 

Extemalizin 
g Problems 

-3 

ADHD; 

years, 11 



(67-C, 72- 


Disruptive 

months 


Problems 

Attention 

C) 


Behavior 



Deficit- 



Disorder, 



(69-B, 66-B) 

Hyperactivit 

Total 


NOS 




y Problems 

Problems 






(72-C, 75- 

(65-C,71- 






C) 

C) 






Oppositiona 

Conduct 






1 Defiant 

Problems 






Problems 

(69-B, 71- 






(66-B, 77- 
C) 

C) 





School 

Activities 

Rule- 





Competence 

Competence 

Breaking 





(30-C, 55-N) 

(55-N, 34- 

Behavior 


Client 4 




B) 

(67-B, 66- 




Total Competence 


B) 


Caucasian 

Male 



(40-B, 46-N) 

Thought 

Problems 

Extemalizin 




Anxious/ 

(59-N, 66- 

g Problems 


ODD; R/0 
ADHD; 

13 years, 4 


Depressed 

B) 

(72-C, 75- 


months - 13 

147 

(72-C, 57-N) 

Aggressive 

C) 

+4 

RnrHprlinp 

years, 9 



Intellectual 

Functioning 

(Clinical 

months 


Withdrawn/ 

Behavior 

Total 




Depressed 
(70-C, 63-N) 

(76-C, 86- 
C) 

Problems 
(72-C, 75- 
C) 


Judgment) 



Social Problems 

Oppositiona 





(72-C, 66-B) 

1 Defiant 

Affective 






Problems 

Problems 





Attention 

(71-C, 80- 

(66-B, 66- 





Problems 

C) 

B) 
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(67-B, 61-N) 

Internalizing 
Problems 
(70-C, 61-B) 

Anxiety Problems 
(73-C, 62-N) 


Attention 
Deficit- 
Hyperactivit 
y Problems 
(68-B, 67- 
B) 

Conduct 
Problems 
(71-C, 69- 
B) 


Client 5 

African 

American 

Male 

Adjustment 
Disorder 
with Mixed 
Disturbance 
s of 

Conduct 

and 

Emotions 

7 years, 3 
months - 7 
years, 7 
months 

118 

Social Competence 
(34-B, 43-N) 

Anxious/Depresse 

d 

(72-C, 59-N) 

Thought Problems 
(67-B, 61-N) 

Aggressive 
Behavior 
(76-C, 66-C) 

Affective 
Problems 
(70-C, 63-N) 

Oppositional 
Defiant Problems 
(73-C, 66-B) 

Activities 
Competence 
(40-N, 30- 
C) 

Rule- 
Breaking 
Behavior 
(67-B, 70- 
C) 

Total 

Competence 
(35-C, 34- 
C) 

Withdrawn 
/ Depressed 
(68-B, 70- 
C) 

Internalizin 
g Problems 
(70-C, 65- 
C) 

Externalizin 
g Problems 
(73-C, 68- 
C) 

Total 
Problems 
(70-C, 65- 
C) 

Conduct 
Problems 
(70-C, 70- 
C) 

+4 

Client 6 

African 

American 

Male 

ADHD; 

ODD; 

8 years, 7 
months - 9 
years, 3 
months 

244 

Activities 
Competence 
(33-B, 38-N) 

Attention 
Problems 
(71-C, 67-B) 

Social 

Competence 
(39-N, 29- 
C) 

Total 

Competence 
(34-C, 28- 

Externalizin 
g Problems 
(71-C, 75- 
C) 

Total 
Problems 
(68-C, 73- 

-10 
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Mood 
Disorder, 
NOS; R/0 
Conduct 
Disorder 





C) 

Anxious/ 
Depressed 
(53-N, 66- 
B) 

Withdrawn/ 
Depressed 
(58-N, 68- 

B) 

Social 
Problems 
(62-N, 75- 

C) 

Thought 
Problems 
(64-N, 70- 
C) 

Rule- 
Breaking 
Behavior 
(64-N, 67- 

B) 

Aggressive 
Behavior 
(72-C, 83- 

C) 

Internalizin 
g Problems 
(54-N, 66- 
C) 

Affective 
Problems 
(52-N, 65- 

B) 

Oppositiona 
1 Defiant 
Problems 
(70-C, 73- 

C) 

Conduct 

Problems 


C) 

Attention 
Deficit- 
Hyperactivit 
y Problems 
(77-C, 77- 
C) 
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(69-B, 76- 
C) 



Client 7 

Bi-Racial 

Male 

ODD; 

ADHD; 

Mixed 

Receptive- 

Expressive 

Language 

Disorder; 

Learning 

Disorder, 

NOS 

9 years, 9 
months - 10 
years, 7 
months 

371 

Withdrawn/ 
Depressed 
(68-B, 62-N) 

Social Problems 
(69-B, 50-N) 

Attention 
Problems 
(79-C, 55-N) 

Rule-Breaking 

Behavior 

(71-C,51-N) 

Aggressive 
Behavior 
(73-C, 50-N) 

Internalizing 
Problems 
(66-C, 54-N) 

Externalizing 
Problems 
(73-C, 40-N) 

Total Problems 
(72-C, 48-N) 

Affective 
Problems 
(65-B, 56-N) 

Anxiety Problems 
(66-B, 50-N) 

Attention Deficit- 
Hyperactivity 
Problems 
(69-B, 50-N) 

Oppositional 
Defiant Problems 
(70-C, 50-N) 

Conduct Problems 
(74-C, 50-N) 

Social 

Competence 
(37-N, 23- 
C) 

School 
Competence 
(27-C, 25- 
C) 

Activities 
Competence 
(30-C, 31- 
N) 

Total 

Competence 
(26-C, 25- 
C) 

+n 
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Client 8 

Caucasian 

Male 

ODD 

3 years, 7 
months - 3 
years, 11 
months 

146 

Emotionally 
Reactive 
(69-B, 62-N) 

Anxious/Depresse 

d 

(70-C, 59-N) 

Withdrawn 
(76-C, 60-N) 

Sleep Problems 
(67-B, 53-N) 

Aggressive 
Behavior 
(82-C, 59-N) 

Internalizing 
Problems 
(71-C, 60-B) 

Externalizing 
Problems 
(74-C, 58-N) 

Affective 
Problems 
(79-C, 67-B) 

Anxiety Problems 
(67-B, 50-N) 

Pervasive 
Developmental 
Problems 
(74-C, 66-B) 

Attention Deficit- 
Hyperactivity 
Problems 
(67-B, 54-N) 

Oppositional 
Defiant Problems 
(77-C, 59-N) 


Total 
Problems 
(71-C, 58- 
N) 

+12 

Client 9 
Caucasian 

12 years, 3 
months - 13 
years, 4 

401 

Activities 
Competence 
(29-C, 40-N) 

Somatic 
Complaints 
(67-B, 70- 


+17 
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Male 

months 


Social Competence 

ADHD; 



(26-C, 35-B) 

ODD; 

Mixed 



School 

Receptive- 



Competence 

Expressive 

Language 



(33-B, 39-N) 

Disorder; 



Total Competence 

Phonologica 



(23-C, 34-C) 

1 Disorder 



Anxious/Depresse 

d 

(76-C, 54-N) 




Withdrawn/ 
Depressed 
(66-B, 50-N) 




Social Problems 
(79-C, 66-B) 




Thought Problems 
(80-C, 63-N) 




Attention 
Problems 
(76-C, 61-N) 




Rule-Breaking 
Behavior 
(66-B, 57-N) 




Aggressive 
Behavior 
(78-C, 58-N) 




Internalizing 
Problems 
(71-C, 58-N) 




Externalizing 
Problems 
(72-C, 58-N) 




Total Problems 
(78-C, 61-B) 




Affective 

Problems 

(77-C,51-N) 


C) 

Somatic 
Problems 
(68-B, 70- 
C) 
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Anxiety Problems 
(74-C, 58-N) 

Attention Deficit- 
Hyperactivity 
Problems 
(77-C, 68-B) 

Oppositional 
Defiant Problems 
(71-C, 55-N) 

Conduct Problems 
(69-B, 56-N) 




Client 10 

African 

American 

Male 

PTSD; 

ODD; 

Mood 

Disorder, 

NOS; 

Bereavemen 
t; Learning 
Disorder, 
NOS 

9 years, 3 
months - 12 
years, 0 
months 

871 

Social Competence 
(34-B, 48-N) 

School 
Competence 
(24-C, 37-N) 

Withdrawn/ 
Depressed 
(70-C, 63-N) 

Thought Problems 
(74-C, 70-C) 

Attention 
Problems 
(75-C, 55-N) 

Rule-Breaking 
Behavior 
(80-C, 62-N) 

Aggressive 
Behavior 
(84-C, 60-N) 

Externalizing 
Problems 
(80-C, 61-B) 

Oppositional 
Defiant Problems 
(73-C, 58-N) 

Conduct Problems 

Activities 
Competence 
(53-N, 28- 
C) 

Somatic 
Complaints 
(64-N, 67- 

B) 

Social 
Problems 
(53-N, 72- 

C) 

Anxiety 
Problems 
(55-N, 66- 
B) 

Somatic 
Problems 
(61-N, 68- 
B) 

Total 

Competence 
(32-C, 31- 
C) 

Internalizin 
g Problems 
(68-C, 65- 
C) 

Total 
Problems 
(75-C, 65- 
C) 

Affective 
Problems 
(76-C, 73- 
C) 

+5 
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(88-C, 64-N) 




Client 11 

African 

American 

Female 

ADHD; 
R/0 Mood 
Disorder 

15 years, 4 
months - 15 
years, 9 
months 

155 

Social Competence 
(35-B, 59-N) 

Total Competence 
(36-C,51-N) 

Thought Problems 
(77-C, 70-C) 

Rule-Breaking 
Behavior 
(69-B, 64-N) 

Anxious/ 
Depressed 
(51-N, 65- 
B) 

Withdrawn/ 
Depressed 
(54-N, 66- 

B) 

Social 
Problems 
(51-N, 73- 

C) 

Attention 
Problems 
(61-N, 66- 

B) 

Internalizin 
g Problems 
(50-N, 66- 

C) 

Total 
Problems 
(65-C, 67- 
C) 

Affective 
Problems 
(60-N, 72- 
C) 

Anxiety 
Problems 
(51-N, 68- 
B) 

Attention 
Deficit- 
Hyperactivit 
y Problems 
(63-N, 66- 
B) 

Extemalizin 
g Problems 
(67-C, 61- 
B) 

Conduct 
Problems 
(68-B, 65- 
B) 

-5 

Client 12 

8 years, 3 
months - 8 

211 

Anxious / 
Depressed 


Social 

Competence 

+6 
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Bi-Racial 

Male 

ODD; 

ADHD 

years, 9 
months 


(67-B, 59-N) 

Social Problems 
(70-C, 67-B) 

Rule-Breaking 
Behavior 
(73-C, 70-C) 

Aggressive 
Behavior 
(91-C, 75-C) 

Oppositional 
Defiant Problems 
(77-C, 66-B) 

Conduct Problems 
(80-C,71-C) 


(34-B, 34- 
B) 

Withdrawn/ 
Depressed 
(66-B, 68- 
B) 

Internalizin 
g Problems 
(66-C, 63- 

B) 

Externalizin 
g Problems 
(79-C, 73- 

C) 

Total 
Problems 
(72-C, 66- 
C) 


Client 13 

Caucasian 

Male 

ADHD; 

ODD; 

Learning 

Disorder, 

NOS; OCD 

(Secondary 

to 

PANDAS); 

Tic 

Disorder 

(Secondary 

to 

PANDAS); 

Enuresis, 

Nocturnal 

12 years, 5 
months - 12 
years, 7 
months 

43 

School 
Competence 
(33-B, 37-N) 

Total Competence 
(38-B, 43-N) 

Affective 
Problems 
(66-B, 61-N) 

Anxious/ 
Depressed 
(60-N, 66- 
B) 

Attention 
Problems 
(61-N, 67- 

B) 

Anxiety 
Problems 
(66-B, 71- 

C) 

Attention 
Deficit- 
Hyperactivit 
y Problems 
(62-N, 70- 
C) 

Thought 
Problems 
(72-C, 74- 
C) 

Aggressive 
Behavior 
(63-N, 66- 
B) 

Internalizin 
g Problems 
(61-B, 61- 
B) 

Externalizin 
g Problems 
(60-B, 63- 

B) 

Total 
Problems 
(64-C, 67- 

C) 

-1 

Client 14 

Caucasian 

Male 

7 years, 9 
months - 8 
years, 0 
months 

100 

School 
Competence 
(27-C, 29-C) 

Total Competence 

Anxious/ 
Depressed 
(67-B, 74- 
C) 

Withdrawn/ 
Depressed 
(76-C, 76- 
C) 

-6 
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ADHD; 

Adjustment 

Disorder; 

ODD 



(36-C, 39-B) 
Attention 

Deficit/Hyperactivi 
ty Problems 
(75-C, 72-C) 

Somatic 
Complaints 
(53-N, 68- 
B) 

Social 
Problems 
(62-N, 67- 

B) 

Attention 
Problems 
(71-C, 88- 

C) 

Rule- 
Breaking 
Behaviors 
(67-B, 74- 
C) 

Affective 
Problems 
(65-B, 70- 
C) 

Anxiety 
Problems 
(65-B, 72- 
C) 

Somatic 
Problems 
(50-N, 70- 
C) 

Conduct 
Problems 
(65-B, 70- 
C) 

Thought 
Problems 
(70-C, 73- 
C) 

Aggressive 
Behavior 
(66-B, 65- 

B) 

Internalizin 
g Problems 
(70-C, 75- 

C) 

Extemalizin 
g Problems 
(67-C,71- 
C) 

Total 
Problems 
(72-C, 75- 
C) 

Oppositiona 
1 Defiant 
Problems 
(70-C, 70- 
C) 


Client 15 

Caucasian 

Male 

ADHD; 

R/0 

Learning 

Disorder 

6 years, 11 
months - 7 
years, 8 
months 

238 

Attention 
Problems 
(71-C, 67-B) 

Attention Deficit- 
Hyperactivity 
Problems 
(72-C, 69-B) 

Total 

Competence 
(49-N, 39- 
B) 

Total 
Problems 
(56-N, 61- 
B) 

-Hi 
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Social Competence 
(34-B, 45-N) 






Total Competence 
(35-C, 44-N) 






Thought Problems 
(67-B, 58-N) 






Attention 
Problems 
(79-C, 66-B) 


School 

Client 16 

African 

American 



Rule-Breaking 
Behavior 
(73-C, 67-B) 


Competence 
(27-C, 27- 
C) 

Male 

8 years, 0 
months - 9 

673 

Aggressive 
Behavior 
(84-C, 65-B) 


Total 

Problems 

ADHD; 

ODD; 

years, 10 
months 


(72-C, 60- 
B) 

Learning 

Disorder, 



Externalizing 
Problems 
(74-C, 66-C) 


Affective 

NOS 




Problems 
(55-B, 56- 




Attention 

Deficit/Hyperactivi 
ty Problems 
(80-C, 62-N) 


N) 




Oppositional 
Defiant Problems 
(80-C, 62-N) 






Conduct Problems 
(75-C, 63-N) 




TABLE 2, Next Page! 


+10 
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Table 2: Clinically Significant Change Based on CBCL Scales 

Conduct Problems J 
Oppositionol Defiant Problems , 

Attention Dclicit/Hyperactivity Problems , 

Somatic Problems _ 

Anxiety Problems ^ 

Affective Problems ^ 

Total Problems ! 

Cxternaluinit Problems 
Internalizing Problems _ 

Aggressive Behavior 
Rule-Breaking Behaviors ^ 

Attention Problems ^ 

Thought Problems ^ 

Social Problems 
Somatic Complaints 
Withdrawn/Depressed ^ 

Anxious/Oepressed ^ 

Total Competence ! 

School Competence ^ 

Social Competence • 

Activities Competence 

O 

Results 

Using the reliable change index, the authors found that ten of the sixteen participants (62.5%) 
make clinically significant progress over the course of the average 278 days. On the subscales, excluding 
the composite scores, there were a potential of 270 possible scale changes of which 174 scales changed 
across all subjects (64.4%). Looking at those subscales, 89 scales clinically improved (51.1% scale 
improvement) and 56 scales clinically regressed (31.1%). The remainder of the scales showed no 
clinically significant change. Of the composite scales, there was a potential for 60 composite changes. 
Fifty-eight changes actually occurred. Of those 58 changes, 18 were significantly improved (31%), 7 
clinically regressed (12.1%), and 33 showed no clinically significant change (56.9%). Examining the 
children who improved compared to those who regressed, the children improved, on average, on 7.3 net 
scales. Those who regressed did so, on average, on 4.8 net scales. 

Of the 232 scales showing pathology observed in this present study, 93 (40.0%) show recovery. 
Of those 93, 71 (76.3%) showed a full recovery and 22 (23.7%) were partially recovered. 


1 RoRroiilon 
UnchonRcd 
1 Progress 


10 


12 


14 
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On the individual scales, more children progressed than regressed with regards to the following: 
conduct problems, oppositional defiant problems, attention deficit/hyperactivity problems, externalizing 
problems, aggressive behavior, rule-breaking behavior, attention problems, withdrawn/depressed 
symptoms, total competence, school competence, and school competence. The scales for conduct 
problems, thought problems, and externalizing problems, though, largely remained unchanged. 

Regression occurred more often than not with regards to internalizing problems, social problems, 
somatic complaints, and activities competence. Because only one child in this present study was 
administered the CBCL/1.5-5, results on that standardized assessment are limited to Client 8. Reliable 
change on those scales can be found in Table 1. 

With regards to gender, nine (approximately 64%) of the males showed progress and one (50%) 
of the females showed progress. With regards to race, four of the seven (approximately 57%) African 
American children showed progress and four of the seven (approximately 57%) Caucasian children 
showed progress. Both (100%) of the bi-racial children showed progress. With regards primary 
diagnoses, four of the ten (40%) diagnosed with ADHD showed clinically significant progress; all of the 
children diagnosed with ODD showed clinically significant progress; both the child diagnosed with 
adjustment disorder with mixed disturbance of conduct and emotion and the child diagnosed with PTSD 
made clinically significant progress. It is important to note that these statistics only apply to primary 
diagnoses. Most children (87.5%) had co-morbid diagnoses. 

When the treatment integrity was checked, 23 (30.7%) were BSC progress notes, 11 (14.7%) 
were MT progress notes, and 41 (54.7%) were TSS progress notes. Seventy-one (94.6%) directly 
reflected the goals and interventions written by the BSC in the treatment plan. It is clear that treatment 
integrity was maintained and it is reasonable to attribute at least part of reliable change to the BHRS 
program. 


Discussion 

This was the first evaluation of BHRS for children with EBD published to our knowledge. It is 
clear from our data that children can make substantial gains from the program. Ten of the sixteen 
participants in this study made clinically significant progress. The average number of days between the 
pre- and post-scores was 278 days. It appears that 62.5% of the children show progress within one year. 
Notwithstanding, it does appear that the longer a child is in the system, the better their outcomes are. 

Race and diagnosis does not appear to matter with regards to success in the program. Regarding subscale 
progress, 51.1% show improvement to the 31.1% that regress. When examining the composite scales, 
31% improved while 12.1% regressed. Regression should be closely monitored by supervisors so that the 
BSC is afforded the opportunity to revise their treatment plan under the guidance of those with, 
potentially, more expertise and experience to design more effective interventions. Forty percent of the 
scales observed show a recovery, 76.3% of which show a full recovery. Finally, it appears that once 
progress is obtained on a particular subscale, it is easier for change to occur on the subsequent scales. 

This provides evidence that targeting the appropriate behavior can lead to a wider range of behavior 
change. 


Specific to BATP, it appears that the staff needs additional training and supervision with regards 
to internalizing problems, social problems, somatic complaints, and activities competence. There is a 
behavior model of depression treatment outlined by Fewinsohn, Clarke, and Hops (1990) in which a 
functional analysis of depression is conducted. Currently, the staff is being trained in the model. For the 
supervisors at BATP, this means implementing more training programs in this model. 
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While the method of analysis of treatment integrity used in this report has problems with it, overall, the 
treatment integrity is high. Future studies need to focus on a better by which of assessing treatment 
integrity, such as direct observation. 

While behavior analytic interventions have a strong research based supporting them for the 
treatment of EBD diagnoses (O’Donohue & Ferguson, 2006), this is the first study that shows that they 
can be effectively in Pennsylvania’ s BFIRS format. This study can be looked as the first step in 
developing programs for children with EBD. Eight percent of the student population suffers from an 
EBD while only one percent receives additional services (full, 2008). In Pennsylvania, BHRS is one of 
the options teachers and parents should consider as promising for the treatment of EBD. Still, much more 
research needs to be done. Problems such as poor school adjustment, delinquent behaviors, adult 
psychopathology, disruptions to the academic process, impediments to social functioning, unsafe 
environments, lower grades, failed courses, and high drop-out rates directly affect the future. They affect 
social and economic outcomes, unemployment, criminality, substance abuse, and aberrant sexual 
behavior (Sacks & Kerns, 2008). Interventions need to occur early in the child’s development to improve 
life for the future. 

Eimitations to this study included a small sample size, one instrument upon which to base change, 
no control group, no randomization, and its inherent A-B experimental design. With the small sample 
size in this present study, it was difficult to obtain any statistical significance with regards to change. It 
was for this reason that the authors chose to rely on clinical significance. Also, because this study was a 
program evaluation for BATP, no control group was included to refute the null hypothesis that BHRS has 
no effect on behavioral change. In these types of studies, the instrument’ s stander error of the measure 
serves as the control for how effective change is (Jacobson, Eollette, & Revenstorf, 1984). Since the 
parents are not program staff, they represent an independent evaluation of the program which is a strength 
of this evaluation process. In addition, staying away from statistical change and talking about the 
percentage of children making improvement is much more customer friendly for parents of children who 
seeking services. Ideally, we envision a world in which every parent entering a child know the 
percentage of children progressing to normal levels based on the statistical instruments used in that clinic. 
In the future, the authors would like to eliminate these limitations by including more participants which is 
directly related to the cases available for services in Montgomery County, Pennsylvania. Also, BATP 
evaluators have begun to use more behavior assessment instruments in the evaluation process. We 
believe that this process will give a more in depth look at the specific progress our children are making 
and lead to continuous improvement through the feedback to the staff of the child’s areas of progress 
during the supervisory process. The focus of this study lay predominately in problem behaviors and an 
increase in adaptive behaviors was not included. Euture, follow-up studies hope to include this facet of an 
improved quality of life. Einally, the authors would like to include a control group in the future to prove 
that BHRS provides causality for behavioral change. 
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